Pandas Profiling Report

Dataset statistics

Number of variables	19
Number of observations	8760687
Missing cells	17521350
Missing cells (%)	10.5%
Duplicate rows	0
Duplicate rows (%)	0.0%
Total size in memory	1.2 GiB
Average record size in memory	152.0 B

Variable types

Categorical	5
DateTime	2
Numeric	11
Boolean	1

Alerts

`congestion_surcharge` has constant value "2.5"	Constant
`airport_fee` has constant value "0.0"	Constant
`trip_distance` is highly correlated with `fare_amount` and 1 other fields	High correlation
`payment_type` is highly correlated with `tip_amount`	High correlation
`fare_amount` is highly correlated with `trip_distance` and 1 other fields	High correlation
`tip_amount` is highly correlated with `payment_type` and 1 other fields	High correlation
`total_amount` is highly correlated with `trip_distance` and 2 other fields	High correlation
`fare_amount` is highly correlated with `tip_amount` and 1 other fields	High correlation
`mta_tax` is highly correlated with `improvement_surcharge`	High correlation
`tip_amount` is highly correlated with `fare_amount` and 1 other fields	High correlation
`tolls_amount` is highly correlated with `total_amount`	High correlation
`improvement_surcharge` is highly correlated with `mta_tax`	High correlation
`total_amount` is highly correlated with `fare_amount` and 2 other fields	High correlation
`trip_distance` is highly correlated with `fare_amount` and 1 other fields	High correlation
`payment_type` is highly correlated with `tip_amount`	High correlation
`fare_amount` is highly correlated with `trip_distance` and 1 other fields	High correlation
`tip_amount` is highly correlated with `payment_type`	High correlation
`total_amount` is highly correlated with `trip_distance` and 1 other fields	High correlation
`improvement_surcharge` is highly correlated with `airport_fee` and 1 other fields	High correlation
`airport_fee` is highly correlated with `improvement_surcharge` and 4 other fields	High correlation
`congestion_surcharge` is highly correlated with `improvement_surcharge` and 4 other fields	High correlation
`payment_type` is highly correlated with `airport_fee` and 1 other fields	High correlation
`store_and_fwd_flag` is highly correlated with `airport_fee` and 1 other fields	High correlation
`VendorID` is highly correlated with `airport_fee` and 1 other fields	High correlation
`fare_amount` is highly correlated with `extra` and 2 other fields	High correlation
`extra` is highly correlated with `fare_amount` and 2 other fields	High correlation
`mta_tax` is highly correlated with `fare_amount` and 2 other fields	High correlation
`total_amount` is highly correlated with `fare_amount` and 2 other fields	High correlation
`congestion_surcharge` has 8760675 (> 99.9%) missing values	Missing
`airport_fee` has 8760675 (> 99.9%) missing values	Missing
`trip_distance` is highly skewed (γ1 = 2945.565755)	Skewed
`RatecodeID` is highly skewed (γ1 = 131.785505)	Skewed
`fare_amount` is highly skewed (γ1 = 61.39737964)	Skewed
`mta_tax` is highly skewed (γ1 = 141.0589211)	Skewed
`tolls_amount` is highly skewed (γ1 = 131.6560327)	Skewed
`total_amount` is highly skewed (γ1 = 35.98422193)	Skewed
`extra` has 4744786 (54.2%) zeros	Zeros
`tip_amount` has 2895926 (33.1%) zeros	Zeros
`tolls_amount` has 8335744 (95.1%) zeros	Zeros

Reproduction

Analysis started	2022-07-18 19:07:00.712163
Analysis finished	2022-07-18 19:38:26.227214
Duration	31 minutes and 25.52 seconds
Software version	pandas-profiling v3.2.0
Download configuration	config.json

VendorID
Categorical

HIGH CORRELATION

Distinct	2
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Memory size	66.8 MiB

2	4914553
1	3846134

Length

Max length	1
Median length	1
Mean length	1
Min length	1

Characters and Unicode

Total characters	8760687
Distinct characters	2
Distinct categories	1 ?
Distinct scripts	1 ?
Distinct blocks	1 ?

The Unicode Standard assigns character properties to each code point, which can be used to analyse textual variables.

Unique

Unique	0 ?
Unique (%)	0.0%

Sample

1st row	1
2nd row	1
3rd row	1
4th row	1
5th row	1

Common Values

Value	Count	Frequency (%)
2	4914553	56.1%
1	3846134	43.9%

Length

Histogram of lengths of the category

Category Frequency Plot

Value	Count	Frequency (%)
2	4914553	56.1%
1	3846134	43.9%

Most occurring characters

Value	Count	Frequency (%)
2	4914553	56.1%
1	3846134	43.9%

Most occurring categories

Value	Count	Frequency (%)
Decimal Number	8760687	100.0%

Most frequent character per category

Decimal Number

Value	Count	Frequency (%)
2	4914553	56.1%
1	3846134	43.9%

Most occurring scripts

Value	Count	Frequency (%)
Common	8760687	100.0%

Most frequent character per script

Common

Value	Count	Frequency (%)
2	4914553	56.1%
1	3846134	43.9%

Most occurring blocks

Value	Count	Frequency (%)
ASCII	8760687	100.0%

Most frequent character per block

ASCII

Value	Count	Frequency (%)
2	4914553	56.1%
1	3846134	43.9%

tpep_pickup_datetime
Date

Distinct	2311532
Distinct (%)	26.4%
Missing	0
Missing (%)	0.0%
Memory size	66.8 MiB

Minimum	2001-01-05 11:45:23
Maximum	2018-07-27 04:06:37

Histogram

Histogram with fixed size bins (bins=50)

tpep_dropoff_datetime
Date

Distinct	2315089
Distinct (%)	26.4%
Missing	0
Missing (%)	0.0%
Memory size	66.8 MiB

Minimum	2001-01-05 11:52:05
Maximum	2018-07-27 04:46:57

Histogram

Histogram with fixed size bins (bins=50)

passenger_count
Real number (ℝ_≥0)

Distinct	10
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%
Mean	1.606807434

Minimum	0
Maximum	9
Zeros	59269
Zeros (%)	0.7%
Negative	0
Negative (%)	0.0%
Memory size	66.8 MiB

Quantile statistics

Minimum	0
5-th percentile	1
Q1	1
median	1
Q3	2
95-th percentile	5
Maximum	9
Range	9
Interquartile range (IQR)	1

Descriptive statistics

Standard deviation	1.258420269
Coefficient of variation (CV)	0.7831805118
Kurtosis	4.110614297
Mean	1.606807434
Median Absolute Deviation (MAD)	0
Skewness	2.236753458
Sum	14076737
Variance	1.583621573
Monotonicity	Not monotonic

Histogram with fixed size bins (bins=10)

Value	Count	Frequency (%)
1	6249564	71.3%
2	1271678	14.5%
5	414473	4.7%
3	351928	4.0%
6	250115	2.9%
4	163572	1.9%
0	59269	0.7%
7	39	< 0.1%
9	25	< 0.1%
8	24	< 0.1%

Minimum 10 values
Maximum 10 values

Value	Count	Frequency (%)
0	59269	0.7%
1	6249564	71.3%
2	1271678	14.5%
3	351928	4.0%
4	163572	1.9%
5	414473	4.7%
6	250115	2.9%
7	39	< 0.1%
8	24	< 0.1%
9	25	< 0.1%

Value	Count	Frequency (%)
9	25	< 0.1%
8	24	< 0.1%
7	39	< 0.1%
6	250115	2.9%
5	414473	4.7%
4	163572	1.9%
3	351928	4.0%
2	1271678	14.5%
1	6249564	71.3%
0	59269	0.7%

trip_distance
Real number (ℝ_≥0)

HIGH CORRELATION
HIGH CORRELATION
SKEWED

Distinct	4397
Distinct (%)	0.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%
Mean	2.804022101

Minimum	0
Maximum	189483.84
Zeros	55377
Zeros (%)	0.6%
Negative	0
Negative (%)	0.0%
Memory size	66.8 MiB

Quantile statistics

Minimum	0
5-th percentile	0.47
Q1	0.91
median	1.55
Q3	2.84
95-th percentile	10.31
Maximum	189483.84
Range	189483.84
Interquartile range (IQR)	1.93

Descriptive statistics

Standard deviation	64.12049747
Coefficient of variation (CV)	22.86732955
Kurtosis	8704373.201
Mean	2.804022101
Median Absolute Deviation (MAD)	0.75
Skewness	2945.565755
Sum	24565159.97
Variance	4111.438195
Monotonicity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
0.8	212156	2.4%
0.9	210312	2.4%
0.7	204209	2.3%
1	203818	2.3%
1.1	191421	2.2%
0.6	184251	2.1%
1.2	179719	2.1%
1.3	167783	1.9%
1.4	155003	1.8%
0.5	150137	1.7%
Other values (4387)	6901878	78.8%

Minimum 10 values
Maximum 10 values

Value	Count	Frequency (%)
0	55377	0.6%
0.01	2660	< 0.1%
0.02	1963	< 0.1%
0.03	1656	< 0.1%
0.04	1356	< 0.1%
0.05	1153	< 0.1%
0.06	1063	< 0.1%
0.07	991	< 0.1%
0.08	925	< 0.1%
0.09	894	< 0.1%

Value	Count	Frequency (%)
189483.84	1	< 0.1%
830.8	1	< 0.1%
484.91	1	< 0.1%
267.7	1	< 0.1%
252.1	1	< 0.1%
132.61	1	< 0.1%
130.46	1	< 0.1%
128.73	1	< 0.1%
127.06	1	< 0.1%
123.3	1	< 0.1%

RatecodeID
Real number (ℝ_≥0)

SKEWED

Distinct	7
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%
Mean	1.0395453

Minimum	1
Maximum	99
Zeros	0
Zeros (%)	0.0%
Negative	0
Negative (%)	0.0%
Memory size	66.8 MiB

Quantile statistics

Minimum	1
5-th percentile	1
Q1	1
median	1
Q3	1
95-th percentile	1
Maximum	99
Range	98
Interquartile range (IQR)	0

Descriptive statistics

Standard deviation	0.4450619326
Coefficient of variation (CV)	0.4281313497
Kurtosis	28417.31093
Mean	1.0395453
Median Absolute Deviation (MAD)	0
Skewness	131.785505
Sum	9107131
Variance	0.1980801239
Monotonicity	Not monotonic

Histogram with fixed size bins (bins=7)

Value	Count	Frequency (%)
1	8533295	97.4%
2	179273	2.0%
5	27735	0.3%
3	15129	0.2%
4	5080	0.1%
99	106	< 0.1%
6	69	< 0.1%

Minimum 10 values
Maximum 10 values

Value	Count	Frequency (%)
1	8533295	97.4%
2	179273	2.0%
3	15129	0.2%
4	5080	0.1%
5	27735	0.3%
6	69	< 0.1%
99	106	< 0.1%

Value	Count	Frequency (%)
99	106	< 0.1%
6	69	< 0.1%
5	27735	0.3%
4	5080	0.1%
3	15129	0.2%
2	179273	2.0%
1	8533295	97.4%

store_and_fwd_flag
Boolean

HIGH CORRELATION

Distinct	2
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Memory size	8.4 MiB

False	8729886
True	30801

Common Values
Category Frequency Plot

Value	Count	Frequency (%)
False	8729886	99.6%
True	30801	0.4%

PULocationID
Real number (ℝ_≥0)

Distinct	259
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%
Mean	164.4579091

Minimum	1
Maximum	265
Zeros	0
Zeros (%)	0.0%
Negative	0
Negative (%)	0.0%
Memory size	66.8 MiB

Quantile statistics

Minimum	1
5-th percentile	48
Q1	116
median	162
Q3	234
95-th percentile	262
Maximum	265
Range	264
Interquartile range (IQR)	118

Descriptive statistics

Standard deviation	66.35990085
Coefficient of variation (CV)	0.4035068987
Kurtosis	-0.8874974706
Mean	164.4579091
Median Absolute Deviation (MAD)	67
Skewness	-0.2792103357
Sum	1440764266
Variance	4403.63644
Monotonicity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
237	361012	4.1%
161	354958	4.1%
236	345712	3.9%
230	309228	3.5%
162	308339	3.5%
186	292545	3.3%
234	283990	3.2%
170	277931	3.2%
48	270624	3.1%
142	264095	3.0%
Other values (249)	5692253	65.0%

Minimum 10 values
Maximum 10 values

Value	Count	Frequency (%)
1	571	< 0.1%
2	4	< 0.1%
3	37	< 0.1%
4	19656	0.2%
5	2	< 0.1%
6	51	< 0.1%
7	14673	0.2%
8	71	< 0.1%
9	46	< 0.1%
10	2079	< 0.1%

Value	Count	Frequency (%)
265	3817	< 0.1%
264	155094	1.8%
263	170014	1.9%
262	109309	1.2%
261	47424	0.5%
260	5308	0.1%
259	63	< 0.1%
258	98	< 0.1%
257	256	< 0.1%
256	6546	0.1%

DOLocationID
Real number (ℝ_≥0)

Distinct	261
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%
Mean	162.7269672

Minimum	1
Maximum	265
Zeros	0
Zeros (%)	0.0%
Negative	0
Negative (%)	0.0%
Memory size	66.8 MiB

Quantile statistics

Minimum	1
5-th percentile	43
Q1	113
median	162
Q3	234
95-th percentile	262
Maximum	265
Range	264
Interquartile range (IQR)	121

Descriptive statistics

Standard deviation	70.31144507
Coefficient of variation (CV)	0.4320823173
Kurtosis	-0.9421178723
Mean	162.7269672
Median Absolute Deviation (MAD)	68
Skewness	-0.3373784234
Sum	1425600026
Variance	4943.699308
Monotonicity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
236	360459	4.1%
161	337453	3.9%
237	320277	3.7%
170	279155	3.2%
230	270526	3.1%
162	267091	3.0%
234	245764	2.8%
142	239840	2.7%
48	234355	2.7%
239	222967	2.5%
Other values (251)	5982800	68.3%

Minimum 10 values
Maximum 10 values

Value	Count	Frequency (%)
1	14044	0.2%
2	5	< 0.1%
3	515	< 0.1%
4	42568	0.5%
5	55	< 0.1%
6	167	< 0.1%
7	38223	0.4%
8	130	< 0.1%
9	450	< 0.1%
10	3086	< 0.1%

Value	Count	Frequency (%)
265	19413	0.2%
264	141297	1.6%
263	173597	2.0%
262	118036	1.3%
261	40265	0.5%
260	9728	0.1%
259	846	< 0.1%
258	1400	< 0.1%
257	4209	< 0.1%
256	20113	0.2%

payment_type
Categorical

HIGH CORRELATION
HIGH CORRELATION
HIGH CORRELATION

Distinct	4
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Memory size	66.8 MiB

1	6106416
2	2599215
3	43204
4	11852

Length

Max length	1
Median length	1
Mean length	1
Min length	1

Characters and Unicode

Total characters	8760687
Distinct characters	4
Distinct categories	1 ?
Distinct scripts	1 ?
Distinct blocks	1 ?

The Unicode Standard assigns character properties to each code point, which can be used to analyse textual variables.

Unique

Unique	0 ?
Unique (%)	0.0%

Sample

1st row	2
2nd row	2
3rd row	1
4th row	2
5th row	1

Common Values

Value	Count	Frequency (%)
1	6106416	69.7%
2	2599215	29.7%
3	43204	0.5%
4	11852	0.1%

Length

Histogram of lengths of the category

Category Frequency Plot

Value	Count	Frequency (%)
1	6106416	69.7%
2	2599215	29.7%
3	43204	0.5%
4	11852	0.1%

Most occurring characters

Value	Count	Frequency (%)
1	6106416	69.7%
2	2599215	29.7%
3	43204	0.5%
4	11852	0.1%

Most occurring categories

Value	Count	Frequency (%)
Decimal Number	8760687	100.0%

Most frequent character per category

Decimal Number

Value	Count	Frequency (%)
1	6106416	69.7%
2	2599215	29.7%
3	43204	0.5%
4	11852	0.1%

Most occurring scripts

Value	Count	Frequency (%)
Common	8760687	100.0%

Most frequent character per script

Common

Value	Count	Frequency (%)
1	6106416	69.7%
2	2599215	29.7%
3	43204	0.5%
4	11852	0.1%

Most occurring blocks

Value	Count	Frequency (%)
ASCII	8760687	100.0%

Most frequent character per block

ASCII

Value	Count	Frequency (%)
1	6106416	69.7%
2	2599215	29.7%
3	43204	0.5%
4	11852	0.1%

fare_amount
Real number (ℝ)

HIGH CORRELATION
HIGH CORRELATION
HIGH CORRELATION
HIGH CORRELATION
SKEWED

Distinct	1714
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%
Mean	12.244426

Minimum	-450
Maximum	8016
Zeros	2238
Zeros (%)	< 0.1%
Negative	4260
Negative (%)	< 0.1%
Memory size	66.8 MiB

Quantile statistics

Minimum	-450
5-th percentile	4
Q1	6
median	9
Q3	13.5
95-th percentile	34
Maximum	8016
Range	8466
Interquartile range (IQR)	7.5

Descriptive statistics

Standard deviation	11.68320695
Coefficient of variation (CV)	0.954165344
Kurtosis	33432.27545
Mean	12.244426
Median Absolute Deviation (MAD)	3
Skewness	61.39737964
Sum	107269583.7
Variance	136.4973245
Monotonicity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
6	473270	5.4%
5.5	465207	5.3%
6.5	461959	5.3%
7	446414	5.1%
5	433292	4.9%
7.5	421895	4.8%
8	399185	4.6%
8.5	369251	4.2%
4.5	359667	4.1%
9	342286	3.9%
Other values (1704)	4588261	52.4%

Minimum 10 values
Maximum 10 values

Value	Count	Frequency (%)
-450	1	< 0.1%
-430	1	< 0.1%
-420	1	< 0.1%
-231	1	< 0.1%
-207	1	< 0.1%
-200	1	< 0.1%
-198	2	< 0.1%
-180	1	< 0.1%
-160	1	< 0.1%
-150	2	< 0.1%

Value	Count	Frequency (%)
8016	1	< 0.1%
5724.5	1	< 0.1%
3013.5	1	< 0.1%
3006	1	< 0.1%
2693	1	< 0.1%
2409	1	< 0.1%
2309.5	1	< 0.1%
1018.5	1	< 0.1%
1016.5	1	< 0.1%
980.06	1	< 0.1%

extra
Real number (ℝ)

HIGH CORRELATION
ZEROS

Distinct	42
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%
Mean	0.3246882465

Minimum	-44.69
Maximum	60
Zeros	4744786
Zeros (%)	54.2%
Negative	2106
Negative (%)	< 0.1%
Memory size	66.8 MiB

Quantile statistics

Minimum	-44.69
5-th percentile	0
Q1	0
median	0
Q3	0.5
95-th percentile	1
Maximum	60
Range	104.69
Interquartile range (IQR)	0.5

Descriptive statistics

Standard deviation	0.4502554665
Coefficient of variation (CV)	1.386731646
Kurtosis	74.60443447
Mean	0.3246882465
Median Absolute Deviation (MAD)	0
Skewness	3.361137354
Sum	2844492.1
Variance	0.2027299852
Monotonicity	Not monotonic

Histogram with fixed size bins (bins=42)

Value	Count	Frequency (%)
0	4744786	54.2%
0.5	2549099	29.1%
1	1433472	16.4%
4.5	30512	0.3%
-0.5	1484	< 0.1%
-1	572	< 0.1%
0.3	380	< 0.1%
1.3	178	< 0.1%
0.8	89	< 0.1%
-4.5	43	< 0.1%
Other values (32)	72	< 0.1%

Minimum 10 values
Maximum 10 values

Value	Count	Frequency (%)
-44.69	1	< 0.1%
-9.61	1	< 0.1%
-5.53	1	< 0.1%
-4.5	43	< 0.1%
-1	572	< 0.1%
-0.5	1484	< 0.1%
-0.49	2	< 0.1%
-0.4	1	< 0.1%
-0.35	1	< 0.1%
0	4744786	54.2%

Value	Count	Frequency (%)
60	1	< 0.1%
30.5	1	< 0.1%
25	1	< 0.1%
19	1	< 0.1%
18	1	< 0.1%
15.55	1	< 0.1%
10.5	1	< 0.1%
6.52	2	< 0.1%
4.8	8	< 0.1%
4.54	3	< 0.1%

mta_tax
Real number (ℝ)

HIGH CORRELATION
HIGH CORRELATION
SKEWED

Distinct	15
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%
Mean	0.4975066111

Minimum	-0.5
Maximum	45.49
Zeros	35938
Zeros (%)	0.4%
Negative	4153
Negative (%)	< 0.1%
Memory size	66.8 MiB

Quantile statistics

Minimum	-0.5
5-th percentile	0.5
Q1	0.5
median	0.5
Q3	0.5
95-th percentile	0.5
Maximum	45.49
Range	45.99
Interquartile range (IQR)	0

Descriptive statistics

Standard deviation	0.04333281101
Coefficient of variation (CV)	0.08709997022
Kurtosis	144286.3777
Mean	0.4975066111
Median Absolute Deviation (MAD)	0
Skewness	141.0589211
Sum	4358499.7
Variance	0.00187773251
Monotonicity	Not monotonic

Histogram with fixed size bins (bins=15)

Value	Count	Frequency (%)
0.5	8720505	99.5%
0	35938	0.4%
-0.5	4153	< 0.1%
3	73	< 0.1%
0.01	5	< 0.1%
0.35	4	< 0.1%
0.9	1	< 0.1%
0.4	1	< 0.1%
6.33	1	< 0.1%
15	1	< 0.1%
Other values (5)	5	< 0.1%

Minimum 10 values
Maximum 10 values

Value	Count	Frequency (%)
-0.5	4153	< 0.1%
0	35938	0.4%
0.01	5	< 0.1%
0.32	1	< 0.1%
0.35	4	< 0.1%
0.4	1	< 0.1%
0.5	8720505	99.5%
0.6	1	< 0.1%
0.9	1	< 0.1%
3	73	< 0.1%

Value	Count	Frequency (%)
45.49	1	< 0.1%
23.8	1	< 0.1%
15	1	< 0.1%
10.41	1	< 0.1%
6.33	1	< 0.1%
3	73	< 0.1%
0.9	1	< 0.1%
0.6	1	< 0.1%
0.5	8720505	99.5%
0.4	1	< 0.1%

tip_amount
Real number (ℝ)

HIGH CORRELATION
HIGH CORRELATION
HIGH CORRELATION
ZEROS

Distinct	3397
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%
Mean	1.818759305

Minimum	-88.8
Maximum	441.71
Zeros	2895926
Zeros (%)	33.1%
Negative	55
Negative (%)	< 0.1%
Memory size	66.8 MiB

Quantile statistics

Minimum	-88.8
5-th percentile	0
Q1	0
median	1.36
Q3	2.35
95-th percentile	6.05
Maximum	441.71
Range	530.51
Interquartile range (IQR)	2.35

Descriptive statistics

Standard deviation	2.486374703
Coefficient of variation (CV)	1.36707188
Kurtosis	599.1830975
Mean	1.818759305
Median Absolute Deviation (MAD)	1.36
Skewness	8.70551644
Sum	15933581
Variance	6.182059164
Monotonicity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
0	2895926	33.1%
1	529907	6.0%
2	271504	3.1%
1.5	108003	1.2%
1.46	96331	1.1%
1.56	95907	1.1%
1.36	94515	1.1%
1.66	93161	1.1%
1.45	90280	1.0%
1.76	89817	1.0%
Other values (3387)	4395336	50.2%

Minimum 10 values
Maximum 10 values

Value	Count	Frequency (%)
-88.8	1	< 0.1%
-13.06	1	< 0.1%
-11	1	< 0.1%
-10	1	< 0.1%
-7.92	1	< 0.1%
-3.32	1	< 0.1%
-3	1	< 0.1%
-2.3	1	< 0.1%
-2	2	< 0.1%
-1.66	1	< 0.1%

Value	Count	Frequency (%)
441.71	1	< 0.1%
415	1	< 0.1%
411	1	< 0.1%
355	1	< 0.1%
330	1	< 0.1%
310	1	< 0.1%
260	2	< 0.1%
250	1	< 0.1%
230	1	< 0.1%
220	2	< 0.1%

tolls_amount
Real number (ℝ)

HIGH CORRELATION
SKEWED
ZEROS

Distinct	967
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%
Mean	0.3026157184

Minimum	-15
Maximum	950.7
Zeros	8335744
Zeros (%)	95.1%
Negative	25
Negative (%)	< 0.1%
Memory size	66.8 MiB

Quantile statistics

Minimum	-15
5-th percentile	0
Q1	0
median	0
Q3	0
95-th percentile	0
Maximum	950.7
Range	965.7
Interquartile range (IQR)	0

Descriptive statistics

Standard deviation	1.738183755
Coefficient of variation (CV)	5.74386474
Kurtosis	55788.09396
Mean	0.3026157184
Median Absolute Deviation (MAD)	0
Skewness	131.6560327
Sum	2651121.59
Variance	3.021282764
Monotonicity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
0	8335744	95.1%
5.76	391626	4.5%
10.5	6018	0.1%
2.64	3870	< 0.1%
12.5	3816	< 0.1%
5.54	2557	< 0.1%
11.52	2083	< 0.1%
17.5	848	< 0.1%
15.5	780	< 0.1%
16.26	620	< 0.1%
Other values (957)	12725	0.1%

Minimum 10 values
Maximum 10 values

Value	Count	Frequency (%)
-15	1	< 0.1%
-12.5	3	< 0.1%
-11.52	1	< 0.1%
-10.5	4	< 0.1%
-5.76	15	< 0.1%
-4	1	< 0.1%
0	8335744	95.1%
0.01	10	< 0.1%
0.02	1	< 0.1%
0.03	1	< 0.1%

Value	Count	Frequency (%)
950.7	1	< 0.1%
910.9	1	< 0.1%
910.5	1	< 0.1%
821	1	< 0.1%
795.76	1	< 0.1%
765.76	1	< 0.1%
600	1	< 0.1%
576	3	< 0.1%
567.56	1	< 0.1%
556.33	1	< 0.1%

improvement_surcharge
Categorical

HIGH CORRELATION
HIGH CORRELATION

Distinct	4
Distinct (%)	< 0.1%
Missing	0
Missing (%)	0.0%
Memory size	66.8 MiB

0.3	8753737
-0.3	4255
0.0	2569
1.0	126

Length

Max length	4
Median length	3
Mean length	3.000485693
Min length	3

Characters and Unicode

Total characters	26286316
Distinct characters	5
Distinct categories	3 ?
Distinct scripts	1 ?
Distinct blocks	1 ?

The Unicode Standard assigns character properties to each code point, which can be used to analyse textual variables.

Unique

Unique	0 ?
Unique (%)	0.0%

Sample

1st row	0.3
2nd row	0.3
3rd row	0.3
4th row	0.3
5th row	0.3

Common Values

Value	Count	Frequency (%)
0.3	8753737	99.9%
-0.3	4255	< 0.1%
0.0	2569	< 0.1%
1.0	126	< 0.1%

Length

Histogram of lengths of the category

Category Frequency Plot

Value	Count	Frequency (%)
0.3	8757992	> 99.9%
0.0	2569	< 0.1%
1.0	126	< 0.1%

Most occurring characters

Value	Count	Frequency (%)
0	8763256	33.3%
.	8760687	33.3%
3	8757992	33.3%
-	4255	< 0.1%
1	126	< 0.1%

Most occurring categories

Value	Count	Frequency (%)
Decimal Number	17521374	66.7%
Other Punctuation	8760687	33.3%
Dash Punctuation	4255	< 0.1%

Most frequent character per category

Decimal Number

Value	Count	Frequency (%)
0	8763256	50.0%
3	8757992	50.0%
1	126	< 0.1%

Other Punctuation

Value	Count	Frequency (%)
.	8760687	100.0%

Dash Punctuation

Value	Count	Frequency (%)
-	4255	100.0%

Most occurring scripts

Value	Count	Frequency (%)
Common	26286316	100.0%

Most frequent character per script

Common

Value	Count	Frequency (%)
0	8763256	33.3%
.	8760687	33.3%
3	8757992	33.3%
-	4255	< 0.1%
1	126	< 0.1%

Most occurring blocks

Value	Count	Frequency (%)
ASCII	26286316	100.0%

Most frequent character per block

ASCII

Value	Count	Frequency (%)
0	8763256	33.3%
.	8760687	33.3%
3	8757992	33.3%
-	4255	< 0.1%
1	126	< 0.1%

total_amount
Real number (ℝ)

HIGH CORRELATION
HIGH CORRELATION
HIGH CORRELATION
HIGH CORRELATION
SKEWED

Distinct	11514
Distinct (%)	0.1%
Missing	0
Missing (%)	0.0%
Infinite	0
Infinite (%)	0.0%
Mean	15.49108833

Minimum	-450.3
Maximum	8016.8
Zeros	1066
Zeros (%)	< 0.1%
Negative	4260
Negative (%)	< 0.1%
Memory size	66.8 MiB

Quantile statistics

Minimum	-450.3
5-th percentile	5.8
Q1	8.3
median	11.3
Q3	16.62
95-th percentile	43.95
Maximum	8016.8
Range	8467.1
Interquartile range (IQR)	8.32

Descriptive statistics

Standard deviation	14.19545671
Coefficient of variation (CV)	0.9163627758
Kurtosis	15348.25133
Mean	15.49108833
Median Absolute Deviation (MAD)	3.5
Skewness	35.98422193
Sum	135712576.2
Variance	201.5109911
Monotonicity	Not monotonic

Histogram with fixed size bins (bins=50)

Value	Count	Frequency (%)
7.3	208335	2.4%
7.8	206427	2.4%
6.8	203105	2.3%
8.3	199717	2.3%
8.8	195497	2.2%
6.3	185976	2.1%
9.3	173507	2.0%
5.8	162675	1.9%
9.8	160903	1.8%
10.3	147321	1.7%
Other values (11504)	6917224	79.0%

Minimum 10 values
Maximum 10 values

Value	Count	Frequency (%)
-450.3	1	< 0.1%
-430.3	1	< 0.1%
-420.8	1	< 0.1%
-231.8	1	< 0.1%
-209.3	1	< 0.1%
-207.3	1	< 0.1%
-200.3	1	< 0.1%
-198.5	1	< 0.1%
-180.8	1	< 0.1%
-160.8	1	< 0.1%

Value	Count	Frequency (%)
8016.8	1	< 0.1%
5726.3	1	< 0.1%
3014.3	1	< 0.1%
3006.8	1	< 0.1%
2694.8	1	< 0.1%
2417.81	1	< 0.1%
2310.3	1	< 0.1%
1019.3	1	< 0.1%
1017.3	1	< 0.1%
1003.5	1	< 0.1%

congestion_surcharge
Categorical

CONSTANT
HIGH CORRELATION
MISSING
REJECTED

Distinct	1
Distinct (%)	8.3%
Missing	8760675
Missing (%)	> 99.9%
Memory size	66.8 MiB

2.5	12

Length

Max length	3
Median length	3
Mean length	3
Min length	3

Characters and Unicode

Total characters	36
Distinct characters	3
Distinct categories	2 ?
Distinct scripts	1 ?
Distinct blocks	1 ?

The Unicode Standard assigns character properties to each code point, which can be used to analyse textual variables.

Unique

Unique	0 ?
Unique (%)	0.0%

Sample

1st row	2.5
2nd row	2.5
3rd row	2.5
4th row	2.5
5th row	2.5

Common Values

Value	Count	Frequency (%)
2.5	12	< 0.1%
(Missing)	8760675	> 99.9%

Length

Histogram of lengths of the category

Category Frequency Plot

Value	Count	Frequency (%)
2.5	12	100.0%

Most occurring characters

Value	Count	Frequency (%)
2	12	33.3%
.	12	33.3%
5	12	33.3%

Most occurring categories

Value	Count	Frequency (%)
Decimal Number	24	66.7%
Other Punctuation	12	33.3%

Most frequent character per category

Decimal Number

Value	Count	Frequency (%)
2	12	50.0%
5	12	50.0%

Other Punctuation

Value	Count	Frequency (%)
.	12	100.0%

Most occurring scripts

Value	Count	Frequency (%)
Common	36	100.0%

Most frequent character per script

Common

Value	Count	Frequency (%)
2	12	33.3%
.	12	33.3%
5	12	33.3%

Most occurring blocks

Value	Count	Frequency (%)
ASCII	36	100.0%

Most frequent character per block

ASCII

Value	Count	Frequency (%)
2	12	33.3%
.	12	33.3%
5	12	33.3%

airport_fee
Categorical

CONSTANT
HIGH CORRELATION
MISSING
REJECTED

Distinct	1
Distinct (%)	8.3%
Missing	8760675
Missing (%)	> 99.9%
Memory size	66.8 MiB

0.0	12

Length

Max length	3
Median length	3
Mean length	3
Min length	3

Characters and Unicode

Total characters	36
Distinct characters	2
Distinct categories	2 ?
Distinct scripts	1 ?
Distinct blocks	1 ?

The Unicode Standard assigns character properties to each code point, which can be used to analyse textual variables.

Unique

Unique	0 ?
Unique (%)	0.0%

Sample

1st row	0.0
2nd row	0.0
3rd row	0.0
4th row	0.0
5th row	0.0

Common Values

Value	Count	Frequency (%)
0.0	12	< 0.1%
(Missing)	8760675	> 99.9%

Length

Histogram of lengths of the category

Category Frequency Plot

Value	Count	Frequency (%)
0.0	12	100.0%

Most occurring characters

Value	Count	Frequency (%)
0	24	66.7%
.	12	33.3%

Most occurring categories

Value	Count	Frequency (%)
Decimal Number	24	66.7%
Other Punctuation	12	33.3%

Most frequent character per category

Decimal Number

Value	Count	Frequency (%)
0	24	100.0%

Other Punctuation

Value	Count	Frequency (%)
.	12	100.0%

Most occurring scripts

Value	Count	Frequency (%)
Common	36	100.0%

Most frequent character per script

Common

Value	Count	Frequency (%)
0	24	66.7%
.	12	33.3%

Most occurring blocks

Value	Count	Frequency (%)
ASCII	36	100.0%

Most frequent character per block

ASCII

Value	Count	Frequency (%)
0	24	66.7%
.	12	33.3%

Spearman's ρ

The Spearman's rank correlation coefficient (ρ) is a measure of monotonic correlation between two variables, and is therefore better in catching nonlinear monotonic correlations than Pearson's r. It's value lies between -1 and +1, -1 indicating total negative monotonic correlation, 0 indicating no monotonic correlation and 1 indicating total positive monotonic correlation.

To calculate ρ for two variables X and Y, one divides the covariance of the rank variables of X and Y by the product of their standard deviations.

Pearson's r

The Pearson's correlation coefficient (r) is a measure of linear correlation between two variables. It's value lies between -1 and +1, -1 indicating total negative linear correlation, 0 indicating no linear correlation and 1 indicating total positive linear correlation. Furthermore, r is invariant under separate changes in location and scale of the two variables, implying that for a linear function the angle to the x-axis does not affect r.

To calculate r for two variables X and Y, one divides the covariance of X and Y by the product of their standard deviations.

Kendall's τ

Similarly to Spearman's rank correlation coefficient, the Kendall rank correlation coefficient (τ) measures ordinal association between two variables. It's value lies between -1 and +1, -1 indicating total negative correlation, 0 indicating no correlation and 1 indicating total positive correlation.

To calculate τ for two variables X and Y, one determines the number of concordant and discordant pairs of observations. τ is given by the number of concordant pairs minus the discordant pairs divided by the total number of pairs.

Cramér's V (φc)

Cramér's V is an association measure for nominal random variables. The coefficient ranges from 0 to 1, with 0 indicating independence and 1 indicating perfect association. The empirical estimators used for Cramér's V have been proved to be biased, even for large samples. We use a bias-corrected measure that has been proposed by Bergsma in 2013 that can be found here.

Phik (φk)

Phik (φk) is a new and practical correlation coefficient that works consistently between categorical, ordinal and interval variables, captures non-linear dependency and reverts to the Pearson correlation coefficient in case of a bivariate normal input distribution. There is extensive documentation available here.

A simple visualization of nullity by column.

Nullity matrix is a data-dense display which lets you quickly visually pick out patterns in data completion.

The correlation heatmap measures nullity correlation: how strongly the presence or absence of one variable affects the presence of another.

The dendrogram allows you to more fully correlate variable completion, revealing trends deeper than the pairwise ones visible in the correlation heatmap.

First rows

	VendorID	tpep_pickup_datetime	tpep_dropoff_datetime	passenger_count	trip_distance	RatecodeID	store_and_fwd_flag	PULocationID	DOLocationID	payment_type	fare_amount	extra	mta_tax	tip_amount	improvement_surcharge	total_amount	congestion_surcharge	airport_fee
0	1	2018-01-01 00:21:05	2018-01-01 00:24:23	1	0.5	1	N	41	24	2	4.5	0.5	0.5	0.00	0.3	5.80	NaN	NaN
1	1	2018-01-01 00:44:55	2018-01-01 01:03:05	1	2.7	1	N	239	140	2	14.0	0.5	0.5	0.00	0.3	15.30	NaN	NaN
2	1	2018-01-01 00:08:26	2018-01-01 00:14:21	2	0.8	1	N	262	141	1	6.0	0.5	0.5	1.00	0.3	8.30	NaN	NaN
3	1	2018-01-01 00:20:22	2018-01-01 00:52:51	1	10.2	1	N	140	257	2	33.5	0.5	0.5	0.00	0.3	34.80	NaN	NaN
4	1	2018-01-01 00:09:18	2018-01-01 00:27:06	2	2.5	1	N	246	239	1	12.5	0.5	0.5	2.75	0.3	16.55	NaN	NaN
5	1	2018-01-01 00:29:29	2018-01-01 00:32:48	3	0.5	1	N	143	143	2	4.5	0.5	0.5	0.00	0.3	5.80	NaN	NaN
6	1	2018-01-01 00:38:08	2018-01-01 00:48:24	2	1.7	1	N	50	239	1	9.0	0.5	0.5	2.05	0.3	12.35	NaN	NaN
7	1	2018-01-01 00:49:29	2018-01-01 00:51:53	1	0.7	1	N	239	238	1	4.0	0.5	0.5	1.00	0.3	6.30	NaN	NaN
8	1	2018-01-01 00:56:38	2018-01-01 01:01:05	1	1.0	1	N	238	24	1	5.5	0.5	0.5	1.70	0.3	8.50	NaN	NaN
9	1	2018-01-01 00:17:04	2018-01-01 00:22:24	1	0.7	1	N	170	170	2	5.5	0.5	0.5	0.00	0.3	6.80	NaN	NaN

Last rows

	VendorID	tpep_pickup_datetime	tpep_dropoff_datetime	passenger_count	trip_distance	RatecodeID	store_and_fwd_flag	PULocationID	DOLocationID	payment_type	fare_amount	extra	mta_tax	tip_amount	improvement_surcharge	total_amount	congestion_surcharge	airport_fee
8760677	1	2018-01-31 23:05:11	2018-01-31 23:13:53	1	2.60	1	N	137	148	2	10.0	0.5	0.5	0.00	0.3	11.30	NaN	NaN
8760678	1	2018-01-31 23:41:11	2018-01-31 23:45:22	1	0.30	1	N	163	230	2	4.5	0.5	0.5	0.00	0.3	5.80	NaN	NaN
8760679	1	2018-01-31 23:52:53	2018-01-31 23:57:32	1	1.20	1	N	43	162	1	6.0	0.5	0.5	1.50	0.3	8.80	NaN	NaN
8760680	2	2018-01-31 23:20:51	2018-01-31 23:30:51	1	1.65	1	N	234	158	1	8.5	0.5	0.5	2.45	0.3	12.25	NaN	NaN
8760681	2	2018-01-31 23:57:33	2018-02-01 00:07:44	1	2.92	1	N	230	238	2	11.0	0.5	0.5	0.00	0.3	12.30	NaN	NaN
8760682	1	2018-01-31 23:21:35	2018-01-31 23:34:20	2	2.80	1	N	158	163	1	12.0	0.5	0.5	2.65	0.3	15.95	NaN	NaN
8760683	1	2018-01-31 23:35:51	2018-01-31 23:38:57	1	0.60	1	N	163	162	1	4.5	0.5	0.5	1.15	0.3	6.95	NaN	NaN
8760684	2	2018-01-31 23:28:00	2018-01-31 23:37:09	1	2.95	1	N	74	69	2	10.5	0.5	0.5	0.00	0.3	11.80	NaN	NaN
8760685	2	2018-01-31 23:24:40	2018-01-31 23:25:28	1	0.00	1	N	7	193	2	0.0	0.0	0.0	0.00	0.0	0.00	NaN	NaN
8760686	2	2018-01-31 23:28:16	2018-01-31 23:28:38	1	0.00	1	N	7	193	2	0.0	0.0	0.0	0.00	0.0	0.00	NaN	NaN

Overview

Variables

Common Values

Length

Category Frequency Plot

Most occurring characters

Most occurring categories

Most frequent character per category

Decimal Number

Most occurring scripts

Most frequent character per script

Common

Most occurring blocks

Most frequent character per block

ASCII

Common Values

Length

Category Frequency Plot

Most occurring characters

Most occurring categories

Most frequent character per category

Decimal Number

Most occurring scripts

Most frequent character per script

Common

Most occurring blocks

Most frequent character per block

ASCII

Common Values

Length

Category Frequency Plot

Most occurring characters

Most occurring categories

Most frequent character per category

Decimal Number

Other Punctuation

Dash Punctuation

Most occurring scripts

Most frequent character per script

Common

Most occurring blocks

Most frequent character per block

ASCII

Common Values

Length

Category Frequency Plot

Most occurring characters

Most occurring categories

Most frequent character per category

Decimal Number

Other Punctuation

Most occurring scripts

Most frequent character per script

Common

Most occurring blocks

Most frequent character per block

ASCII

Common Values

Length

Category Frequency Plot

Most occurring characters

Most occurring categories

Most frequent character per category

Decimal Number

Other Punctuation

Most occurring scripts

Most frequent character per script

Common

Most occurring blocks

Most frequent character per block

ASCII

Interactions

Correlations

Spearman's ρ

Pearson's r

Kendall's τ

Cramér's V (φc)

Phik (φk)

Missing values

Sample